Analysis apparatus, analysis method, and storage medium

ABSTRACT

Provided is an analysis apparatus ( 10 ) including a person extraction unit ( 11 ) that analyzes video data to extract a person, a time calculation unit ( 12 ) that calculates a continuous appearance time period for which the extracted person has been continuously present in a predetermined area and a reappearance time interval until the extracted person reappears in the predetermined area for each extracted person, and an inference unit ( 13 ) that infers a characteristic of the extracted person on the basis of the continuous appearance time period and the reappearance time interval.

The present application is a Continuation Application of U.S. patentapplication Ser. No. 16/080,745 filed on Aug. 29, 2018, which is aNational Stage Entry of international application PCT/JP2017/005100,filed Feb. 13, 2017, which claims the benefit of priority from JapanesePatent Application 2016-067538 filed on Mar. 30, 2016, the disclosuresof all of which are incorporated herein, in their entirety, by thisreference.

TECHNICAL FIELD

The invention relates to an analysis apparatus, an analysis method, anda program.

BACKGROUND ART

The related art is disclosed in Patent Document 1. The Patent Document 1discloses a countermeasure system against suspicious persons whichdetects the face of a person in a captured image of a scene in asurveillance range and determines whether countermeasures are needed orthe degree of countermeasures on the basis of, for example, the size ofthe face, the time for which the person has been continuously present inthe surveillance range, or the number of times the person appears in thesurveillance range. It is assumed that, as the length of time or thenumber of times described above increases, the possibility that theperson is a suspicious person increases.

Patent Documents 2 and 3 disclose an index generation apparatus thatgenerates an index in which a plurality of nodes are hierarchized.

RELATED DOCUMENT Patent Document

[Patent Document 1] Japanese Patent Application Publication No.2006-11728

[Patent Document 2] WO2014/109127

[Patent Document 3] Japanese Patent Application Publication No.2015-49574

SUMMARY OF THE INVENTION Technical Problem

In a case in which, as the time period for which a person has beencontinuously present or the number of times the person appearsincreases, the possibility that the person is a suspicious person isdetermined to be higher as in the technique disclosed in the PatentDocument 1, a determination error is likely to occur. For example, aperson who continuously works in the surveillance range is determined tobe a suspicious person. In order to solve this problem, it is desirableto have various criteria for determination.

An object of the invention is to provide a new technique for inferring acharacteristic of a person extracted from an image.

Solution to Problem

In one exemplary embodiment of the invention, there is provided ananalysis apparatus comprising: a person extraction unit that analyzesvideo data to extract a person; a time calculation unit that calculatesa continuous appearance time period for which the extracted person hasbeen continuously present in a predetermined area and a reappearancetime interval until the extracted person reappears in the predeterminedarea for each extracted person; and an inference unit that infers acharacteristic of the extracted person on the basis of the continuousappearance time period and the reappearance time interval.

In another exemplary embodiment of the invention, there is provided ananalysis method performed by a computer, the method comprising: a personextraction step of analyzing video data to extract a person; a timecalculation step of calculating a continuous appearance time period forwhich the extracted person has been continuously present in apredetermined area and a reappearance time interval until the extractedperson reappears in the predetermined area for each extracted person;and an inference step of inferring a characteristic of the extractedperson on the basis of the continuous appearance time period and thereappearance time interval.

In still another exemplary embodiment of the invention, there isprovided a program that causes a computer to function as: a personextraction unit that analyzes video data to extract a person; a timecalculation unit that calculates a continuous appearance time period forwhich the extracted person has been continuously present in apredetermined area and a reappearance time interval until the extractedperson reappears in the predetermined area for each extracted person;and an inference unit that infers a characteristic of the extractedperson on the basis of the continuous appearance time period and thereappearance time interval.

Advantageous Effects of Invention

According to the invention, it is possible to provide a new techniquefor inferring the characteristic of a person extracted from an image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above objects and other objects, features and advantages will becomemore apparent from the following description of the preferred exampleembodiments and the accompanying drawings below.

FIG. 1 is a conceptual diagram illustrating an example of the hardwareconfiguration of an apparatus according to an exemplary embodiment.

FIG. 2 is an example of a functional block diagram of an analysisapparatus according to the exemplary embodiment.

FIG. 3 is a diagram illustrating an example of a method of calculating acontinuous appearance time period and a reappearance time intervalaccording to the exemplary embodiment.

FIG. 4 is a diagram illustrating an example of the method of calculatingthe continuous appearance time period and the reappearance time intervalaccording to the exemplary embodiment.

FIG. 5 is a diagram illustrating an example of index information thatmay be used in the exemplary embodiment.

FIG. 6 is a diagram schematically illustrating an example of informationhandled by the analysis apparatus according to the exemplary embodiment.

FIG. 7 is a diagram schematically illustrating an example of theinformation handled by the analysis apparatus according to the exemplaryembodiment.

FIG. 8 is a diagram schematically illustrating an example of theinformation handled by the analysis apparatus according to the exemplaryembodiment.

FIG. 9 is a diagram schematically illustrating an example of theinformation handled by the analysis apparatus according to the exemplaryembodiment.

FIG. 10 is a diagram schematically illustrating an example of theinformation handled by the analysis apparatus according to the exemplaryembodiment.

FIG. 11 is a flowchart illustrating an example of the flow of a processof the analysis apparatus according to the exemplary embodiment.

FIG. 12 is an example of the functional block diagram illustrating theanalysis apparatus according to the exemplary embodiment.

FIG. 13 is a diagram schematically illustrating an example of theinformation handled by the analysis apparatus according to the exemplaryembodiment.

FIG. 14 is an example of the functional block diagram illustrating theanalysis apparatus according to the exemplary embodiment.

FIG. 15 is a diagram illustrating an example of a setting screenprovided by the analysis apparatus according to the exemplaryembodiment.

FIG. 16 is a diagram illustrating an example of the setting screenprovided by the analysis apparatus according to the exemplaryembodiment.

DESCRIPTION OF EMBODIMENTS

First, an example of the hardware configuration of an apparatus(analysis apparatus) according to an exemplary embodiment will bedescribed. Each unit of the apparatus according to the exemplaryembodiment is implemented by an arbitrary combination of software andhardware including a central processing unit (CPU), a memory, a programloaded to the memory, a storage unit (which can store a program loadedfrom a storage medium, such as a compact disc (CD), or a server on theInternet, in addition to a program that is stored in the apparatus inadvance in a shipment stage), such as a hard disk storing the program,and an interface for connection to a network of an arbitrary computer.It will be understood by those skilled in the art that there are variousmodification examples of the implementation method and the apparatus.

FIG. 1 is a block diagram illustrating the hardware configuration of theapparatus according to the exemplary embodiment. As illustrated in FIG.1, the apparatus includes a processor 1A, a memory 2A, an input/outputinterface 3A, a peripheral circuit 4A, and a bus 5A. The peripheralcircuit includes various modules.

The bus 5A is a data transmission path through which the processor 1A,the memory 2A, the peripheral circuit 4A, and the input/output interface3A transmit and receive data. The processor 1A is an arithmeticprocessing unit such as a central processing unit (CPU) or a graphicsprocessing unit (GPU). The memory 2A is, for example, a random accessmemory (RAM) or a read only memory (ROM). The input/output interface 3Aincludes, for example, an interface for acquiring information from anexternal apparatus, an external server, and an external sensor. Theprocessor 1A outputs commands to each module and performs calculation onthe basis of the calculation results of the modules.

Next, the exemplary embodiment will be described. The functional blockdiagrams used in the following description of the exemplary embodimentdo not illustrate the structure of each hardware unit, but illustrate afunctional unit block. In the diagrams, each apparatus is implemented byone device. However, a means of implementing each apparatus is notlimited thereto. That is, each apparatus may be physically divided ormay be logically divided. The same components are denoted by the samereference numerals and the description thereof will not be repeated.

First Exemplary Embodiment

First, the outline of this exemplary embodiment will be described. Ananalysis apparatus according to this exemplary embodiment analyzes videodata to extract a person. Then, the analysis apparatus calculates thetime period (continuous appearance time period) for which the extractedperson has been continuously present in a predetermined area and thetime interval (reappearance time interval) until the extracted personreappears in the predetermined area after leaving the predetermined area(disappearing from the predetermined area) for each extracted person.Then, the analysis apparatus infers the characteristic of the extractedperson on the basis of the continuous appearance time period and thereappearance time interval. The characteristic of the extracted personinferred on the basis of the continuous appearance time period and thereappearance time interval is a kind of information that can berecognized from context or the state of the person. Examples of thecharacteristic include a traveler, a passerby, a pickpocket, anoperator, a migrant worker, a suspicious person, a demonstrator, and ahomeless person. Here, the examples are illustrative and are not limitedthereto. Hereinafter, this exemplary embodiment will be described indetail.

FIG. 2 is an example of the functional block diagram of an analysisapparatus 10 according to this exemplary embodiment. As illustrated inFIG. 2, the analysis apparatus 10 includes a person extraction unit 11,a time calculation unit 12, and an inference unit 13.

The person extraction unit 11 analyzes video data and extracts a personfrom the video data. Any technique can be used as a process ofextracting a person.

For example, video data which is captured by one or a plurality ofcameras (for example, surveillance cameras) installed at predeterminedpositions is input to the person extraction unit 11. For example, theperson extraction unit 11 processes the video data in time series andextracts a person from the video data.

The person extraction unit 11 may process all of the frames included inthe video data or process may be performed on a basis of eachpredetermined frame. Then, the person extraction unit 11 extracts aperson from the frame which is a processing target. In addition, theperson extraction unit 11 extracts the feature amount (for example, thefeature amount of the face) of the outward appearance of the extractedperson.

The time calculation unit 12 calculates the continuous appearance timeperiod for which the person extracted by the person extraction unit 11has been continuously present in a predetermined area and thereappearance time interval until the person reappears in thepredetermined area after leaving the predetermined area for eachextracted person. Any technique can be used to calculate the continuousappearance time period and the reappearance time interval. Hereinafter,an example will be described and the invention is not limited thereto.

For example, the time period for which a person appears continuously inthe video data may be the continuous appearance time period and the timeinterval until the person reappears after disappearing from the videodata may be the reappearance time interval. That is, when an n-th frameto be processed is represented by Fn, for example, it is assumed that acertain person is extracted from each of frames F1 to F500, is notextracted from each of frames F501 to F1500, and is extracted from aframe F1501 again. In this case, the time elapsed from the frame F1 tothe frame F500 may be the continuous appearance time period and the timeelapsed from the frame F501 to the frame F1501 may be the reappearancetime interval.

In this case, an area captured by the camera is the predetermined area.For example, the continuous appearance time and the reappearance timeare calculated by the method detailed below, and thus the predeterminedarea can be expanded to the area captured by the camera and a peripheralarea.

Here, it is assumed that video data captured by one camera is aprocessing target. The time calculation unit 12 calculates an elapsedtime t from the extraction of a person A (first person) from the videodata to the next extraction of the person A from the video data. Then,when the elapsed time t is less than a predetermined time t_(s) (amatter of design) (or when the elapsed time t is equal to or less thanthe predetermined time t_(s)), it is determined that the first personhas been continuously present in the predetermined area for the elapsedtime t. On the other hand, when the elapsed time t is equal to orgreater than the predetermined time t_(s) (or when the elapsed time t isgreater than the predetermined time t_(s)), it is determined that thefirst person has not been present in the predetermined area for theelapsed time t.

A detailed example will be described with reference to FIG. 3. Theelapsed time t1 from the first extraction of the person A to the secondextraction of the person A is less than the predetermined time t_(s).Therefore, the time calculation unit 12 determines that the person A hasbeen continuously present in the predetermined area for the time(elapsed time t1) from the first extraction to the second extraction.

An elapsed time t2 from the second extraction of the person A to thethird extraction of the person A is less than the predetermined timet_(s). Therefore, the time calculation unit 12 determines that theperson A has been continuously present in the predetermined area for thetime (elapsed time t2) from the second extraction to the thirdextraction. Then, the time calculation unit 12 determines that theperson A has been continuously present in the predetermined area for thetime (here, the time from the first extraction to the third extract) forwhich the state that an elapsed time is less than the predetermined timet_(s) continues.

An elapsed time t3 from the third extraction of the person A to thefourth extraction of the person A is greater than the predetermined timet_(s). Therefore, the time calculation unit 12 determines that theperson A has not been present in the predetermined area for the time(elapsed time t3) from the third extraction to the fourth extraction.Then, the time calculation unit 12 sets the elapsed time t3 as thereappearance time interval. In addition, the time calculation unit 12sets (t1+t2) as the continuous appearance time period.

Another example will be described. Here, it is assumed that video datacaptured by a plurality of cameras is a processing target. All of theplurality of cameras capture the image of a predetermined position inthe same predetermined area. For example, all of the plurality ofcameras may be installed in the same area of a “∘∘” park. The imagingareas captured by the plurality of cameras may partially overlap eachother or may not overlap each other.

The time calculation unit 12 calculates the elapsed time t from theextraction of the first person from video data captured by a firstcamera to the next extraction of the first person from video data by anycamera (which may be the first camera or another camera). When theelapsed time t is less than the predetermined time t_(s) (a matter ofdesign) (or when the elapsed time t is equal to or less than thepredetermined time t_(s)), it is determined that the first person hasbeen continuously present in the predetermined area for the elapsed timet. On the other hand, when the elapsed time t is equal to or greaterthan the predetermined time t_(s) (or when the elapsed time t is greaterthan the predetermined time t_(s)), it is determined that the firstperson has not been present in the predetermined area for the elapsedtime t.

A detailed example will be described with reference to FIG. 4. It isassumed that the person A is extracted from video data captured by acamera A (Cam A) (first extraction) and is then extracted from videodata captured by a camera B (Cam B) (second extraction). The elapsedtime t1 from the first extraction to the second extraction is less thanthe predetermined time t_(s). Therefore, the time calculation unit 12determines that the person A has been continuously present in thepredetermined area for the time (elapsed time t1) from the firstextraction to the second extraction.

The elapsed time t2 from the second extraction of the person A from thevideo data captured by the camera B (Cam B) to the next (third)extraction of the person A from video data captured by a camera C (CamC) is less than the predetermined time t_(s). Therefore, the timecalculation unit 12 determines that the person A has been continuouslypresent in the predetermined area for the time (elapsed time t2) fromthe second extraction to the third extraction. Then, the timecalculation unit 12 determines that the person A has been continuouslypresent in the predetermined area for the time (the time from the firstextraction to the third extract) for which the state that an elapsedtime is less than the predetermined time t_(s) continues.

The elapsed time t3 from the third extraction of the person A from thevideo data captured by the camera C (Cam C) to the next (fourth)extraction of the person A from the video data captured by the camera B(Cam B) is greater than the predetermined time t_(s). Therefore, thetime calculation unit 12 determines that the person A has not beenpresent in the predetermined area for the time (elapsed time t3) fromthe third extraction to the fourth extraction. Then, the timecalculation unit 12 sets the elapsed time t3 as the reappearance timeinterval. In addition, the time calculation unit 12 sets a time (t1+t2)as the continuous appearance time period.

Next, this exemplary embodiment will be described on the assumption thatthe calculation method described with reference to FIGS. 3 and 4 isused.

Incidentally, it is necessary to determine whether a person extractedfrom a certain frame and a person extracted from the previous frame arethe same person, in order to perform the above-mentioned process. All ofpairs of the feature amounts of the outward appearance of each personextracted from the previous frame and the feature amounts of the outwardappearance of each person extracted from a certain frame may be comparedto perform the determination described above. However, in the case ofthis process, as the accumulated data of persons increases, the numberof pairs to be compared increases and thus processing load increases.Therefore, for example, the method described below may be adopted.

For example, the extracted person may be indexed as illustrated in FIG.5 to determine whether a person is identical to a previously extractedperson by using the index. The use of the index makes it possible toincrease a processing speed. The details of the index and a method forgenerating the index are disclosed in the Patent Documents 2 and 3.Next, the structure of the index illustrated in FIG. 5 and a method forusing the index will be described in brief.

An extraction identifier (ID) “F∘∘∘-∘∘∘∘” illustrated in FIG. 5 isidentification information which is given to each person extracted fromeach frame. “F∘∘∘” is frame identification information and numbersfollowing a hyphen are identification information of each personextracted from each frame. In a case in which the same person isextracted from different frames, different extraction IDs are given tothe person.

In a third layer, nodes corresponding to all of the extraction IDsobtained from the processed frames are arranged. Among a plurality ofnodes arranged in the third layer, nodes with similarity (similaritybetween the feature amounts of the outward appearance) that is equal toor higher than a first level are grouped. In the third layer, aplurality of extraction IDs which are determined to indicate the sameperson are grouped. That is, the first level of the similarity is set toa value that can implement the grouping. Person identificationinformation (person ID) is given so as to correspond to each group inthe third layer.

In a second layer, one node (representative) which is selected from eachof a plurality of groups in the third layer is arranged and isassociated with the group in the third layer. Among a plurality of nodesarranged in the second layer, nodes with similarity that is equal to orhigher than a second level are grouped. The second level of thesimilarity is lower than the first level. That is, the nodes which arenot grouped together on the basis of the first level may be groupedtogether on the basis of the second level.

In a first layer, one node (representative) which is selected from eachof a plurality of groups in the second layer is arranged and isassociated with the group in the second layer.

For example, the time calculation unit 12 indexes a plurality ofextraction IDs obtained by the above-mentioned process as illustrated inFIG. 5.

Then, when a new extraction ID is obtained from a new frame, the timecalculation unit 12 determines whether a person corresponding to theextraction ID is identical to a previously extracted person using theinformation. In addition, the time calculation unit 12 adds the newextraction ID to the index. Next, this process will be described.

First, the time calculation unit 12 sets a plurality of extraction IDsin the first layer as a comparison target. The person extraction unit 11makes a pair of the new extraction ID and each of the plurality ofextraction IDs in the first layer. Then, the person extraction unit 11calculates similarity (similarity between the feature amounts of theoutward appearance) for each pair and determines whether the calculatedsimilarity is equal to or greater than a first threshold value (is equalto or higher than a predetermined level).

In a case in which an extraction ID with similarity that is equal to orgreater than the first threshold value is not present in the firstlayer, the time calculation unit 12 determines that the personcorresponding to the new extraction ID is not identical to anypreviously extracted person. Then, the time calculation unit 12 adds thenew extraction ID to the first to third layers and associates them witheach other. In the second and third layers, a new group is generated bythe added new extraction ID. In addition, a new person ID is issued incorrespondence with the new group in the third layer. Then, the personID is specified as the person ID of the person corresponding to the newextraction ID.

On the other hand, in a case in which the extraction ID with similaritythat is equal to or greater than the first threshold value is present inthe first layer, the time calculation unit 12 changes a comparisontarget to the second layer. Specifically, the group in the second layerassociated with the “extraction ID in the first layer which has beendetermined to have similarity equal to or greater than the firstthreshold value” is set as a comparison target.

Then, the time calculation unit 12 makes a pair of the new extraction IDand each of a plurality of extraction IDs included in the group to beprocessed in the second layer. Then, the time calculation unit 12calculates similarity for each pair and determines whether thecalculated similarity is equal to or greater than a second thresholdvalue. The second threshold value is greater than the first thresholdvalue.

In a case in which an extraction ID with similarity that is equal to orgreater than the second threshold value is not present in the group tobe processed in the second layer, the time calculation unit 12determines that the person corresponding to the new extraction ID is notidentical to any previously extracted person. Then, the time calculationunit 12 adds the new extraction ID to the second and third layers andassociates them with each other. In the second layer, the new extractionID is added to the group to be processed. In the third layer, a newgroup is generated by the added new extraction ID. In addition, a newperson ID is issued in correspondence with the new group in the thirdlayer. Then, the time calculation unit 12 specifies the person ID as theperson ID of the person corresponding to the new extraction ID.

On the other hand, in a case in which the extraction ID with similaritythat is equal to or greater than the second threshold value is presentin the group to be processed in the second layer, the time calculationunit 12 determines that the person corresponding to the new extractionID is identical to a previously extracted person. Then, the timecalculation unit 12 puts the new extraction ID into the group in thethird layer associated with the “extraction ID in the second layer whichhas been determined to have similarity equal to or greater than thesecond threshold value”. In addition, the time calculation unit 12determines the person ID corresponding to the group in the third layeras the person ID of the person corresponding to the new extraction ID.

For example, in this way, it is possible to associate a person ID withone extraction ID or each of a plurality of extraction IDs extractedfrom a new frame.

For example, the time calculation unit 12 may manage informationillustrated in FIG. 6 for each extracted person ID. Then, the timecalculation unit 12 may calculate the continuous appearance time periodand the reappearance time interval, using the information. In theinformation illustrated in FIG. 6, the person ID, the continuousappearance time period, and the latest extraction timing are associatedwith each other.

The values of the continuous appearance time period and the latestextraction timing are updated as needed. For example, when a certainperson is extracted from the video data first and a new person ID isadded to the information illustrated in FIG. 6, “0” is recorded as thecontinuous appearance time period. In addition, the extraction timing isrecorded as the latest extraction timing. The extraction timing isrepresented by, for example, date and time information. The extractiontiming in this example corresponds to, for example, the first extractionillustrated in FIG. 3. Then, the time calculation unit 12 waits for thenext extraction.

Then, in a case in which the person is extracted for the second time, itis determined whether the person has been continuously present for theelapsed time t1 on the basis of the result of the large and smallcomparison between the elapsed time t1 and the predetermined time t_(s),as described above. The elapsed time t1 is calculated on the basis of,for example, the value in the field of latest extraction timing and theextraction timing of the second time. In a case in which it isdetermined that the person has been present, the value of the continuousappearance time period is updated. Specifically, the sum of the valuerecorded at that time and the elapsed time t1 is recorded in the field.Here, t1 (=0+t1) is recorded. Then, the latest extraction timing isupdated to the extraction timing of the second time. Then, the timecalculation unit 12 waits for the next extraction.

Then, in a case in which the person is extracted for the third time, itis determined whether the person has been continuously present for theelapsed time t2 on the basis of the result of the large and smallcomparison between the elapsed time t2 and the predetermined time t_(s),as described above. The elapsed time t2 is calculated on the basis of,for example, the value in the field of latest extraction timing and theextraction timing of the third time. In a case in which it is determinedthat the person has been present, the value of the continuous appearancetime period is updated. Specifically, the sum (t1+t2) of the value (t1)recorded at that time and the elapsed time t2 is recorded in the field.Then, the latest extraction timing is updated to the extraction timingof the third time. Then, the time calculation unit 12 waits for the nextextraction.

Then, in a case in which the person is extracted for the fourth time, itis determined whether the person has been continuously present for theelapsed time t3 on the basis of the result of the large and smallcomparison between the elapsed time t3 and the predetermined time t_(s),as described above. The elapsed time t3 is calculated on the basis of,for example, the value in the field of latest extraction timing and theextraction timing of the fourth time. In a case in which it isdetermined that the person has not been present, the value of thecontinuous appearance time period at that time is fixed as thecontinuous appearance time period of the person. In addition, theelapsed time t3 is fixed as the reappearance time interval of theperson. Then, a pair of the fixed continuous appearance time period andthe fixed reappearance time interval is input to the inference unit 13.

In addition, the value of the continuous appearance time period isupdated. Specifically, “0” is recorded in the field. Then, the latestextraction timing is updated to the extraction timing of the fourthtime. Then, the time calculation unit 12 waits for the next extraction.Then, the same process as described above is repeated.

Returning to FIG. 2, the inference unit 13 infers the characteristic ofthe extracted person on the basis of the continuous appearance timeperiod and the reappearance time interval. The inference unit 13 infersthe characteristic of the person on the basis of the relationshipbetween the continuous appearance time period and the reappearance timeinterval. The inference unit 13 infers the characteristic of the personon the basis of the pair of the continuous appearance time period andthe reappearance time interval input from the time calculation unit 12.

For example, the inference unit 13 may infer the characteristic of theperson (hereinafter, referred to as a personal characteristic in somecases) on the basis of correspondence information (correspondenceinformation indicating the relationship between the continuousappearance time period and the reappearance time interval) in which thepair of the continuous appearance time period and the reappearance timeinterval is associated with the inferred characteristic.

FIG. 7 illustrates an example of the correspondence information. Thecorrespondence information is represented by a graph in which one axis(the horizontal axis in FIG. 7) indicates the continuous appearance timeperiod and the other axis (the vertical axis in FIG. 7) indicates thereappearance time interval. An area on the graph is divided into aplurality of areas and a personal characteristic is associated with eacharea. The divided areas illustrated in FIG. 7 or the personalcharacteristic associated with each area are exemplified as a conceptualdiagram for illustrating the invention and the correspondenceinformation is not limited to the content.

In a case in which the correspondence information is used, the inferenceunit 13 determines which personal characteristic area the pair of thecontinuous appearance time period and the reappearance time interval islocated in, as illustrated in FIG. 7, and infers the personalcharacteristic on the basis of the determination result.

As another example, the inference unit 13 may infer the personalcharacteristic on the basis of the correspondence information havingdifferent contents for each time slot during which a person appears.

That is, as illustrated in FIGS. 8 and 9, the inference unit 13 maystore correspondence information for each time slot. FIG. 8 correspondsto a time slot from 4 a.m. to 10 p.m. and FIG. 9 corresponds to a timeslot from 10 p.m. to 4 a.m. As can be seen from the comparison betweenthe correspondence information items illustrated in FIGS. 8 and 9, thecontents of the correspondence information items are different from eachother. The inference unit 13 may determine one correspondenceinformation item based on the time slot during which the extractedperson appears and may infer the personal characteristic on the basis ofthe determined correspondence information item, using the same method asdescribed above.

For example, the inference unit 13 may use correspondence informationcorresponding to a time slot including a representative timing for theperiod of time for which the extracted person appears. Therepresentative timing may be, for example, the timing (the firstextraction timing in the example illustrated in FIG. 3) when the personis extracted first, the last extraction timing (the third extractiontiming in the example illustrated in FIG. 3), an intermediate timingbetween the first extraction timing and the last extraction timing, orother timings.

In addition, the inference unit 13 may calculate the overlapping periodbetween the time slot corresponding to each correspondence informationitem and the time period for which the person appears. Then, theinference unit 13 may use the correspondence information correspondingto the longer overlapping period. For example, in a case in which theappearance period is from 2 a.m. to 5 a.m., the overlapping periodbetween the time slot (from 4 a.m. to 10 p.m.) corresponding to thecorrespondence information illustrated in FIG. 8 and the appearanceperiod is 1 hour from 4 a.m. to 5 a.m. In contrast, the overlappingperiod between the time slot (from 10 p.m. to 4 a.m.) corresponding tothe correspondence information illustrated in FIG. 9 and the appearanceperiod is 2 hours from 2 a.m. to 4 a.m. In this case, the inference unit13 may use the correspondence information illustrated in FIG. 9.

In the above-mentioned example, two correspondence information itemscorresponding to two time slots are used. However, the number ofcorrespondence information items is a matter of design and is notlimited thereto.

Furthermore, the inference unit 13 may infer the personal characteristicon the basis of the continuous appearance time period, the reappearancetime interval, data indicating a probability distribution which isstored in advance, and the correspondence information.

For example, as illustrated in FIG. 10, the inference unit 13 sets apoint corresponding to the pair of the continuous appearance time periodand the reappearance time interval as a peak position of the probabilitydistribution. Then, the inference unit 13 extracts all personalcharacteristics corresponding to the area including a portion in whichprobability is greater than 0. In addition, the inference unit 13calculates probability corresponding to each personal characteristic onthe basis of the data of the probability distribution. For example, theinference unit 13 may calculate a statistic (for example, a maximumvalue and an intermediate value) in the probability included in eacharea as the probability corresponding to the personal characteristic.

The analysis apparatus 10 may include a notification unit, which is notillustrated in FIG. 2. When an extracted person is inferred to be apredetermined personal characteristic (for example, a suspiciousperson), the notification unit notifies an operator of the person andinformation indicating that the extracted person is inferred to be thepredetermined personal characteristic. The notification can beimplemented through all types of output devices such as a display, anemailer, a speaker, an alarm lamp, and a printer. In the notificationprocess, the notification unit may notify an operator of the image ofthe face of the person.

The analysis apparatus 10 may include a storage unit that stores theinference result of the inference unit 13 and an output unit thatoutputs the inference result, which is not illustrated in FIG. 2. Thestorage unit stores the person ID and the inference result inassociation with each other. The storage unit may store one or aplurality of image data items (image data including the person) inassociation with the person ID. In addition, the storage unit may storethe timing (date and time information) when each person appears.

Then, the output unit may acquire predetermined information from thestorage unit and output the predetermined information, in accordancewith an operation of the operator. For example, when an input specifyingthe personal characteristic is received, a list of the personscorresponding to the personal characteristic may be displayed. Inaddition, when an input specifying the personal characteristic and aperiod is received, a list of the persons who have been inferred to bethe personal characteristic within the period may be displayed. Thedisplay of the list may be implemented, using image data correspondingto each person.

Next, an example of the flow of the process of the analysis apparatus 10according to this exemplary embodiment will be described with referenceto the flowchart illustrated in FIG. 11.

In a person extraction step S10, the person extraction unit 11 analyzesvideo data to extract a person.

In a continuous appearance time period and reappearance time intervalcalculation step S11, the time calculation unit 12 calculates thecontinuous appearance time period for which each person extracted in S10has been continuously present and the reappearance time interval untilthe person reappears in a predetermined area after leaving thepredetermined area.

In a personal characteristic inference step S12, the inference unit 13infers the characteristic of the extracted person on the basis of thecontinuous appearance time period and the reappearance time intervalcalculated in S11.

According to the above-described exemplary embodiment, it is possible toinfer the personal characteristic on the basis of the continuousappearance time period for which a person has been continuously presentin a predetermined area and the reappearance time interval until theperson reappears after leaving the predetermined area. That is, it ispossible to infer the personal characteristic on the basis of newinformation such as the reappearance time interval. Therefore, forexample, the accuracy of inference is expected to be improved and aninference technique is expected to progress.

According to this exemplary embodiment, it is possible to infer thepersonal characteristic on the basis of a pair of the continuousappearance time period and the reappearance time interval. In the caseof this exemplary embodiment, the personal characteristic is inferrednot on the basis of the criterion that “as the continuous appearancetime period increases, the possibility that a person is a suspiciousperson increases”. However, in a case in which the position (a positionin a two-dimensional coordinate illustrated in FIG. 7) of a pair of thevalue of the continuous appearance time period and the value of thereappearance time interval is included in a predetermined range, theperson is inferred to be a certain personal characteristic (for example,a suspicious person). As such, by inferring the personal characteristicon the basis of a plurality of information items (the continuousappearance time period and the reappearance time interval), it ispossible to improve the accuracy of inference.

According to this exemplary embodiment, it is possible to infer thepersonal characteristic on the basis of a plurality of correspondenceinformation items with different contents for each time slot duringwhich a person appears. It is inferred that there is a large differencein personal characteristics between a person who appears during the dayand a person who appears during the night. Since the personalcharacteristic is inferred considering the appearance timing, it ispossible to improve the accuracy of inference.

According to this exemplary embodiment, it is possible to infer thepersonal characteristic, using a probability distribution. The outputresult of the inference unit 13 is just the inference result and is not100 percent accurate. Therefore, there is the possibility that theperson, who is essentially to be inferred as a suspicious person, isinferred as another personal characteristic such as a traveler. Theinference of possible personal characteristics using probabilitydistributions enables a wide inference of possible personalcharacteristics. For example, in the case of the above-mentionedexample, in addition to a traveler, a suspicious person can be inferredas a possible personal characteristic.

According to this exemplary embodiment, the continuous appearance timeperiod and the reappearance time interval can be calculated by themethod described with reference to FIGS. 3 and 4. In the case of thiscalculation method, the area captured by the camera and the periphery ofthe area are set as a predetermined area and the continuous appearancetime period and the reappearance time interval can be calculated for thepredetermined area. That is, it is possible to expand the predeterminedarea to an area which is not captured by the camera.

As a modification example of this exemplary embodiment, in thecorrespondence information, not all pairs of the continuous appearancetime period and the reappearance time interval are necessarilyassociated with personal characteristics as illustrated in, for example,FIG. 7. For example, the correspondence information may include onlysome personal characteristics (for example, a suspicious person and apickpocket) of the information illustrated in FIG. 7. In using suchcorrespondence information, in a case in which the values of thecontinuous appearance time period and the reappearance time intervalcorrespond to the personal characteristics (for example, a suspiciousperson and a pickpocket), the personal characteristic is inferred. Onthe other hand, in a case in which the values of the continuousappearance time period and the reappearance time interval correspond toneither of the personal characteristics (for example, a suspiciousperson and a pickpocket), the person is inferred not to be the personalcharacteristics. This modification example can also be applied to all ofthe exemplary embodiments described below.

Second Exemplary Embodiment

An analysis apparatus 10 according to this exemplary embodiment storesthe personal characteristic inferred by the method described in thefirst exemplary embodiment in association with each person. In a case inwhich a certain person appears repeatedly in a predetermined area, theanalysis apparatus 10 calculates the continuous appearance time periodand the reappearance time interval whenever the person appears. On allsuch occasion, on the basis of the calculation result, a personalcharacteristic is inferred. On the basis of the inference result, theanalysis apparatus 10 counts the number of times each personalcharacteristic is inferred, for each person. Then, the analysisapparatus 10 calculates the reliability of each inferred personalcharacteristic on the basis of the number of counts. Hereinafter, thisexemplary embodiment will be described in detail.

FIG. 12 illustrates an example of the functional block diagram of theanalysis apparatus 10 according to this exemplary embodiment. Asillustrated in FIG. 12, the analysis apparatus 10 includes a personextraction unit 11, a time calculation unit 12, an inference unit 13, acount unit 14, and a reliability calculation unit 15. The analysisapparatus 10 may further include the notification unit, the storageunit, and the output unit described in the first exemplary embodiment,which is not illustrated. The person extraction unit 11, the timecalculation unit 12, the inference unit 13, the notification unit, thestorage unit, and the output unit have the same structure as those inthe first exemplary embodiment.

The count unit 14 counts the number of times each personalcharacteristic which is inferred in correspondence with each person isinferred.

For example, the count unit 14 manages information illustrated in FIG.13. In the information illustrated in FIG. 13, a person ID is associatedwith the number of times each personal characteristic is inferred. Thecount unit 14 updates the information on the basis of the inferenceresult of the inference unit 13.

Whenever a certain person appears repeatedly in a predetermined area,the time calculation unit 12 calculates the continuous appearance timeperiod and the reappearance time interval. On all such occasion, theinference unit 13 infers the personal characteristic on the basis of thecontinuous appearance time period and the reappearance time intervalcalculated whenever the certain person appears repeatedly in thepredetermined area.

The count unit 14 updates the information illustrated in FIG. 13 on thebasis of the result inferred by the inference unit 13 in such a way.

The reliability calculation unit 15 calculates the reliability of theinferred personal characteristic on the basis of the number of timeseach personal characteristic which is inferred in correspondence with acertain person is inferred. The larger the number of times of inferenceis, the higher the reliability is calculated by the reliabilitycalculation unit 15.

The output unit may output predetermined information on the basis of theinference result of the inference unit 13 and the calculation result ofthe reliability calculation unit 15. The output unit can output thepredetermined information in accordance with an operation of theoperator.

For example, when an input specifying a personal characteristic andreliability conditions (for example, reliability is equal to or higherthan a predetermined level) is received, a list of the persons who areinferred to be the personal characteristic with reliability equal to orhigher than the predetermined level may be displayed. The display of thelist may be implemented using image data corresponding to each person.

According to the above-described exemplary embodiment, it is possible toobtain the same advantageous effect as that in the first exemplaryembodiment. In addition, it is possible to calculate the reliability ofeach inferred personal characteristic on the basis of many inferenceresults which are stored in correspondence with each person. As aresult, according to this exemplary embodiment, it is possible toimprove the accuracy of inferring the personal characteristic of eachperson.

Third Exemplary Embodiment

An analysis apparatus 10 according to this exemplary embodiment providesa function of setting correspondence information in which a pair of thecontinuous appearance time period and the reappearance time interval isassociated with a personal characteristic.

FIG. 14 illustrates an example of the functional block diagram of theanalysis apparatus 10 according to this exemplary embodiment. Asillustrated in FIG. 14, the analysis apparatus 10 includes a personextraction unit 11, a time calculation unit 12, an inference unit 13,and a setting unit 16. The analysis apparatus 10 may further include acount unit 14, a reliability calculation unit 15, a notification unit, astorage unit, and an output unit, which are not illustrated in FIG. 14.The person extraction unit 11, the time calculation unit 12, the countunit 14, the reliability calculation unit 15, the notification unit, thestorage unit, and the output unit have the same structure as those inthe first and second exemplary embodiments.

The setting unit 16 has a function of setting the correspondenceinformation in which a pair of the continuous appearance time period andthe reappearance time interval is associated with a personalcharacteristic. The setting unit 16 can set the correspondenceinformation in accordance with an input from the user.

For example, the setting unit 16 may output a setting screen illustratedin FIG. 15 through an output device such as a display. The settingscreen is a screen for receiving input of the name of a personalcharacteristic, the start time and end time of the continuous appearancetime period, and the start time and end time of the reappearance timeinterval.

In addition, the setting unit 16 may output a setting screen illustratedin FIG. 16 through an output device such as a display. The settingscreen is a screen for receiving specification of a predetermined areaon a graph in which one axis (the horizontal axis in FIG. 16) indicatesthe continuous appearance time period and the other axis (the verticalaxis in FIG. 16) indicates the reappearance time interval and an inputof the name of the personal characteristic corresponding to the area.

The inference unit 13 infers a personal characteristic on the basis ofthe correspondence information set by the setting unit 16. The otherstructures of the inference unit 13 are the same as those in the firstand second exemplary embodiments.

According to the above-described exemplary embodiment, it is possible toobtain the same advantageous effect as that in the first and secondexemplary embodiments. In addition, it is possible to freely set variouspersonal characteristics. By setting a personal characteristic of aperson to be detected in the video data, it is possible to detect aperson with the personal characteristic.

Hereinafter, an example of reference exemplary embodiments will beadditionally described.

1. An analysis apparatus including: a person extraction unit thatanalyzes video data to extract a person; a time calculation unit thatcalculates a continuous appearance time period for which the extractedperson has been continuously present in a predetermined area and areappearance time interval until the extracted person reappears in thepredetermined area for each extracted person; and an inference unit thatinfers a characteristic of the extracted person on the basis of thecontinuous appearance time period and the reappearance time interval.

2. The analysis apparatus described in 1, in which the inference unitinfers the characteristic of the person on the basis of a relationshipbetween the continuous appearance time period and the reappearance timeinterval.

3. The analysis apparatus described in 1 or 2 further including: a countunit that counts the number of times each characteristic which isinferred in correspondence with each person is inferred; and areliability calculation unit that calculates reliability of the inferredcharacteristic on the basis of the number of times each characteristicwhich is inferred in correspondence with a certain person is inferred.

4. The analysis apparatus described in any one of 1 to 3, in which theinference unit infers the characteristic of the person on the basis ofcorrespondence information in which a pair of the continuous appearancetime period and the reappearance time interval is associated with acharacteristic.

5. The analysis apparatus described in 4, in which the inference unitinfers the characteristic of the person on the basis of thecorrespondence information having different contents for each time slotduring which the person appears.

6. The analysis apparatus described in 4 or 5, in which the inferenceunit infers the characteristic of the person on the basis of thecontinuous appearance time period, the reappearance time interval, aprobability distribution, and the correspondence information.

7. The analysis apparatus described in any one of 1 to 6,

in which, in a case in which a time t elapsed from the extraction of afirst person from the video data to the next extraction of the firstperson from the video data is less than a predetermined time t_(s), thetime calculation unit determines that the first person has beencontinuously present in the predetermined area for the elapsed time t,and

in a case in which the elapsed time t is equal to or greater than thepredetermined time t_(s), the time calculation unit determines that thefirst person has not been present in the predetermined area for theelapsed time t.

8. An analysis method performed by a computer including: a personextraction step of analyzing video data to extract a person; a timecalculation step of calculating a continuous appearance time period forwhich the extracted person has been continuously present in apredetermined area and a reappearance time interval until the extractedperson reappears in the predetermined area for each extracted person;and an inference step of inferring a characteristic of the extractedperson on the basis of the continuous appearance time period and thereappearance time interval.

8-2. The analysis method described in 8, in which in the inference step,the characteristic of the person is inferred on the basis of arelationship between the continuous appearance time period and thereappearance time interval.

8-3. The analysis method performed by the computer described in 8 or8-2, the method further including: a count step of counting the numberof times each characteristic which is inferred in correspondence witheach person is inferred; and a reliability calculation step ofcalculating reliability of the inferred characteristic on the basis ofthe number of times each characteristic which is inferred incorrespondence with a certain person is inferred.

8-4. The analysis method described in any one of 8 to 8-3, in which inthe inference step, the characteristic of the person is inferred on thebasis of correspondence information in which a pair of the continuousappearance time period and the reappearance time interval is associatedwith a characteristic.

8-5. The analysis method described in 8-4, in which in the inferencestep, the characteristic of the person is inferred on the basis of thecorrespondence information having different contents for each time slotduring which the person appears.

8-6. The analysis method described in 8-4 or 8-5, in which in theinference step, the characteristic of the person is inferred on thebasis of the continuous appearance time period, the reappearance timeinterval, a probability distribution, and the correspondenceinformation.

8-7. The analysis method described in any one of 8 to 8-6, in which inthe time calculation step,

in a case in which a time t elapsed from the extraction of a firstperson from the video data to the next extraction of the first personfrom the video data is less than a predetermined time t_(s), it isdetermined that the first person has been continuously present in thepredetermined area for the elapsed time t, and

in a case in which the elapsed time t is equal to or greater than thepredetermined time t_(s), it is determined that the first person has notbeen present in the predetermined area for the elapsed time t.

9. A program causing a computer to function as: a person extraction unitthat analyzes video data to extract a person; a time calculation unitthat calculates a continuous appearance time period for which theextracted person has been continuously present in a predetermined areaand a reappearance time interval until the extracted person reappears inthe predetermined area for each extracted person; and an inference unitthat infers a characteristic of the extracted person on the basis of thecontinuous appearance time period and the reappearance time interval.

9-2. The program described in 9, in which the inference unit infers thecharacteristic of the person on the basis of a relationship between thecontinuous appearance time period and the reappearance time interval.

9-3. The program described in 9 or 9-2 causing the computer to furtherfunction as: a count unit that counts the number of times eachcharacteristic which is inferred in correspondence with each person isinferred; and a reliability calculation unit that calculates reliabilityof the inferred characteristic on the basis of the number of times eachcharacteristic which is inferred in correspondence with a certain personis inferred.

9-4. The program described in any one of 9 to 9-3, in which theinference unit infers the characteristic of the person on the basis ofcorrespondence information in which a pair of the continuous appearancetime period and the reappearance time interval is associated with acharacteristic.

9-5. The program described in 9-4, in which the inference unit infersthe characteristic of the person on the basis of the correspondenceinformation having different contents for each time slot during whichthe person appears.

9-6. The program described in 9-4 or 9-5, in which the inference unitinfers the characteristic of the person on the basis of the continuousappearance time period, the reappearance time interval, a probabilitydistribution, and the correspondence information.

9-7. The program described in any one of 9 to 9-6,

in which, in a case in which a time t elapsed from the extraction of afirst person from the video data to the next extraction of the firstperson from the video data is less than a predetermined time t_(s), thetime calculation unit determines that the first person has beencontinuously present in the predetermined area for the elapsed time t,and

in a case in which the elapsed time t is equal to or greater than thepredetermined time t_(s), the time calculation unit determines that thefirst person has not been present in the predetermined area for theelapsed time t.

It is apparent that the present invention is not limited to the aboveexemplary embodiment, and may be modified and changed without departingfrom the scope and spirit of the invention.

This application claims priority based on Japanese Patent ApplicationNo. 2016-067538 filed on Mar. 30, 2016, the disclosure of which isincorporated herein in its entirety.

1. An analysis apparatus comprising a memory storing instructions; andat least one processor executing the instructions to perform: outputtinga setting screen to accept, from a user, an input of a correspondencebetween a time period, a time interval, and a personal characteristics,wherein the time period is a time period for which a person has beencontinuously present in a predetermined area in an image, and the timeinterval is a time interval between first point in time when the persondisappears from the predetermined area and second point in time when theperson reappears to the predetermined area.
 2. The analysis apparatusaccording to claim 1, wherein the at least one processor furtherexecutes the instructions to perform: accepting, from a user, an inputof the name of the personal characteristics, start time and end time ofthe time period, and start time and end time of the time interval. 3.The analysis apparatus according to claim 1, wherein the at least oneprocessor further executes the instructions to perform: accepting, froma user, specification of an area on a graph in which one axis indicatesthe time period and the other axis indicates the time interval andaccepting, from a user, an input of the name of the personalcharacteristic corresponding to the area.
 4. The analysis apparatusaccording to claim 1, wherein the at least one processor furtherexecutes the instructions to perform: extracting a person from a videodata; calculating the time period for which the extracted person hasbeen continuously present in the predetermined area and the timeinterval between first point in time when the extracted persondisappears from the predetermined area and second point in time when theextracted person reappears to the predetermined area; and inferring thepersonal characteristics of the extracted person based on the calculatedtime period and the calculated time interval by using thecorrespondence.
 5. An analysis method performed by a computer, themethod comprising: outputting a setting screen to accept, from a user,an input of a correspondence between a time period, a time interval, anda personal characteristics, wherein the time period is a time period forwhich a person has been continuously present in a predetermined area inan image, and the time interval is a time interval between first pointin time when the person disappears from the predetermined area andsecond point in time when the person reappears to the predeterminedarea.
 6. The analysis method according to claim 5, further comprising:accepting, from a user, an input of the name of the personalcharacteristics, start time and end time of the time period, and starttime and end time of the time interval.
 7. The analysis method accordingto claim 5, further comprising: accepting, from a user, specification ofan area on a graph in which one axis indicates the time period and theother axis indicates the time interval and accepting, from a user, aninput of the name of the personal characteristic corresponding to thearea.
 8. The analysis method according to claim 5, further comprising:extracting a person from a video data; calculating the time period forwhich the extracted person has been continuously present in thepredetermined area and the time interval between first point in timewhen the extracted person disappears from the predetermined area andsecond point in time when the extracted person reappears to thepredetermined area; and inferring the personal characteristics of theextracted person based on the calculated time period and the calculatedtime interval by using the correspondence.
 9. A non-transitory computerreadable recording medium storing programs, the programs causing acomputer to perform: outputting a setting screen to accept, from a user,an input of a correspondence between a time period, a time interval, anda personal characteristics, wherein the time period is a time period forwhich a person has been continuously present in a predetermined area inan image, and the time interval is a time interval between first pointin time when the person disappears from the predetermined area andsecond point in time when the person reappears to the predeterminedarea.
 10. The non-transitory computer readable recording mediumaccording to claim 9, wherein the programs causes the computer toperform: accepting, from a user, an input of the name of the personalcharacteristics, start time and end time of the time period, and starttime and end time of the time interval.
 11. The non-transitory computerreadable recording medium according to claim 9, wherein the programscauses the computer to perform: accepting, from a user, specification ofan area on a graph in which one axis indicates the time period and theother axis indicates the time interval and accepting, from a user, aninput of the name of the personal characteristic corresponding to thearea.
 12. The non-transitory computer readable recording mediumaccording to claim 9, wherein the programs causes the computer toperform: extracting a person from a video data; calculating the timeperiod for which the extracted person has been continuously present inthe predetermined area and the time interval between first point in timewhen the extracted person disappears from the predetermined area andsecond point in time when the extracted person reappears to thepredetermined area; and inferring the personal characteristics of theextracted person based on the calculated time period and the calculatedtime interval by using the correspondence.